In Practice News

Follow this topic

Bookmark

Record learning outcomes

ChatGPT 3.5 knows less about therapeutics than pharmacy students, suggests new research from the US.

In a study carried out at the University of Arizona’s R Ken Coit College of Pharmacy, researchers asked the AI bot 210 questions from six pharmacy exams. Overall, the pharmacy students correctly answered 82 per cent of the questions, while ChatGPT answered just 53 per cent correctly. 

The pharmacy students out-performed ChatGPT on questions about gastrointestinal, insomnia, vaginal and vulvovaginal conditions (79 per cent and 33 per cent correct respectively); dermatologic, ophthalmic and otic therapeutics (78 per cent and 39 per cent respectively); pain, fever and tobacco cessation (78 per cent and 34 per cent respectively); cardiology (88 per cent and 69 per cent respectively); neurology (82 per cent and 69 per cent respectively), and critical care (84 per cent and 73 per cent respectively).

ChatGPT was less likely to answer application-based questions correctly than those requiring factual recall (44 per cent and 80 per cent respectively). It was also less likely to correctly answer case-based than other types of questions (45 per cent and 74 per cent respectively).

The AI bot answered 59 per cent of multiple-choice questions correctly. 

“The kinds of places where evidence is limited and judgement is required, which is often [the case] in a clinical setting, was where we found the technology somewhat lacking,” said Brian Erstad, the college’s interim Dean. “Ironically, those are the kinds of questions clinicians are always facing.”